Skip to content

Preserve extra Arrow field definitions during weak schema validation - #298

Closed
pengpengyi92 wants to merge 1 commit into
Point72:mainfrom
pengpengyi92:codex/preserve-arrow-extra-fields-20261114
Closed

pengpengyi92 wants to merge 1 commit into
Point72:mainfrom
pengpengyi92:codex/preserve-arrow-extra-fields-20261114

Conversation

@pengpengyi92

Copy link
Copy Markdown
Contributor

Description

When validating an ArrowTable[ArrowSchema.make(schema, strict=False)], extra columns are kept, but their field definitions are not: appending them by name drops field metadata and resets nullable=False to True.

For example, an extra pa.field("d", pa.int64(), nullable=False, metadata={b"unit": b"shares"}) becomes a nullable field with no metadata after validation. The values remain present, so value-only assertions do not detect the loss.

Pass the original pa.Field to append_column instead. Required columns still cast to the declared schema; extra column order and values, filter mode, and strict rejection remain unchanged. No dependency, public API, or documentation-contract change.

Regression coverage includes multiple metadata-bearing extras, a nullable column containing null, non-nullability, empty record batches, and multi-batch input with required-column casting. Schema checks explicitly include metadata. This is separate from #295, which preserves table-level metadata in the annotation utility.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / code cleanup
  • CI / build configuration
  • Other (describe below)

Validation

On base 83f2b0ac7128c08c92f2b6fd60f58723ac47635a, the new cases fail on metadata/non-nullability loss, including both empty and chunked inputs. With this one-line fix:

python -m pytest ccflow/tests/exttypes/test_arrow.py ccflow/tests/utils/test_arrow.py ccflow/tests/test_arrow.py ccflow/tests/result/test_pyarrow.py ccflow/tests/exttypes/test_narwhals.py ccflow/tests/result/test_narwhals.py ccflow/tests/models/test_narwhals.py -q -o addopts= -p no:cacheprovider --import-mode=importlib --tb=short
python -m ruff check ccflow
python -m ruff format --check ccflow
  • Pydantic 2.14.3: 125 passed, 8 subtests passed.
  • Pydantic 2.6.4: 125 passed, 8 subtests passed, with two namespace warnings.
  • Python 3.12.14, PyArrow 26.0.1, Polars 1.44.1, Narwhals 2.28.0, Windows.
  • Python lint and format checks pass across ccflow (208 files).
  • These are overlapping compatibility runs, not 250 distinct tests. Full make test, documentation lint and performance measurements were not run locally.

Checklist

  • Linting passes (make lint; Python portion passed, docs lint not run)
  • Tests pass (make test; scoped native tests passed as above)
  • Regression tests added
  • Documentation updated (not applicable: no new behavior or API)
  • Changelog / version bump (not included)

AI assistance: Codex assisted with investigation, the original patch, regression tests, and self-review. No separate human or maintainer approval is claimed. The contributor authorized DCO sign-off for this original contribution after verification; no private code or data is included.

Thanks for reviewing; happy to adjust this to the preferred schema-preservation contract.

Signed-off-by: Pengyi Peng <pengpengyi92@users.noreply.github.com>
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.85%. Comparing base (83f2b0a) to head (077cf18).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #298   +/-   ##
=======================================
  Coverage   93.85%   93.85%           
=======================================
  Files         205      205           
  Lines       22944    22974   +30     
  Branches     1511     1512    +1     
=======================================
+ Hits        21533    21563   +30     
  Misses       1120     1120           
  Partials      291      291           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timkpaine

Copy link
Copy Markdown
Member

Thanks for the PR, i've consolidated several open ones in #299

@timkpaine timkpaine closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants